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ABSTRACT 

In this paper, the effectiveness of a "CyberBuch," a 
multimedia program for reading authentic German texts, is assessed in 
three areas. First, based on user evaluation of the visual interface 
design, the usability of the program is assessed with particular 
regard to user reaction to the multimedia components of the program. 
Second, learner behavior while using the program is tracked, and the 
types of multimedia links chosen are tracked. Third, the 
effectiveness of the program for two levels of language learning is 
assessed: short-term recall of vocabulary items and overall reading 
comprehension. Based on answers to questionnaires, users felt 
overwhelmingly that seeing a photograph or movie helped reinforce 
learning, but that the text and audio links were not as helpful. The 
user logs indicated that a greater proportion of picture and movie 
links were chosen than text lii:ks. The vocabulary tests show that 
vocabulary words were learned and recalled better when "defined" with 
picture or movie links than when only textual definitions were 
available. Eight figures and one table illustrate data. 
(Author/MAS) 
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Abstract: In this paper, we assess the effectiveness of a multimedia program for 
reading autlientic German texts in three areas. First, based on user evaluation of the 
visual interface design, we assessed the usability of the program with particular 
regard to user reaction to the multimedia components of the program. Secondly, we 
tracked learner behavior while using tlie program and recorded the types of 
multimedia links that were chosen. Tliirdly, we assessed the effectiveness of tlie 
program for two levels of language learning: short-temi recall of vocabulary items 
and overall reading comprehension. Based on answers to questionnaires, users felt 
overwhelmingly tliat seeing a photograph or movie helped reinforce learning but tliat 
the text and audio links were not as helpful. The user logs indicated that a greater 
proportion of picture and movie links were chosen than text links. The vocabulary 
tests show tliat vocabulary words were learned and recalled better when "defined** 
with picture or movie links than when only textual definitions were available. 



Based on current research and tlieories of reading comprehension, in order for understanding and learning 
to lake place, a reader must first activate prior knowledge, must then reorganize lliese existing knowledge 
structures or schemata, and then can link them to the new material to be learned (cf. Carlson 1990, Bemliardt 
1991). Our multimedia program for the Macintosh computer aids in this process, first, by providing a 
contextualized overview or preview of each reading text in the form of a QuickTime movie clip, which is 
designed to activate in tlie learner's mind the text's global concepts. Second, by avoiding simple translations of 
unknown words and plrrases in the text and instead providing other types of information in the form of 
hypermedia links, such as grapliics (pictures), video (QT movies) and sound, in addition to the definitions of 
words, the learner is encouraged to infer the meaning of tlie word or plirase. These integrated types of 
information help readers formulate their own cognitive schemata, and research has shown that students leani 
more when they "make llieir own comiections" rather than having comiections made for them (or in this case, 
having U^anslations given to them, cf. Hulstijn 1992). The interactive nanire of the software allows learners to 
take an active role in tlie process, i.e. iJiey can choose tlie type of information or media link tliat is most helpful 
to them. At the same lime» the program provides individualized options and self -pacing ability, in order to 
accommodate different learning styles. In addition. Increasing students' motivation to leani is the goal of any 
software but one wliich multimedia applications seem particularly successful at because they address multiple 
senses and thus reinforce learning. 



Key Program Features of CyberBuch 

CyberBuch is a multimedia program whose prime instnictional goal is to improve tlic comprehension skills 
of students who are beginning to read authentic Gennan texts. The metaphor of a library inU-oduccs the user to 
the program, and from tlicre, tlic following key features are accessible: 
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1. Vorschau **Prcview'*: Tliis component is an advance organizer, a meiuis of conveying global iiifonnation 
about the lext» activating top-down processing strategies of first understanding macro-level ideas before micro - 
level details. 

2. Nutivc Speaker Reading the Text {•*Text-Movie Link*'): Accompanying the written text is tlic option of 
hearing the text read by native speakers. Students may listen to liie entire text or select any sentence tlicy vvrnit 
to hear. They control tlie playback as one would a tape. 

3. Multiple Media Links: Within the text, certain words are marked as "hotspots" or *'hot text** and are linked 
to various kinds of information in the form of text» still graphics or videos, For each word, up to three different 
links are available. These "multiple 'media links*' are accessed by dragging the word and dropping it on one of 
the three media icons. In addition to the media link selected, the word is also spoken as a defaiilt. 

Effectiveness Testing 

Researchers today have moved away from the traditional comparative research (medium A vs. medium B, 
CALL (computer- assisted language learning) vs. non-CALL). Rather* more descriptive and evaluative studies 
are being conducted, among other purposes* to acquire "new theoretical infonnation about the psycholinguisiic 
namre of language learning and the way in which a specific computer capability of delivering instruction -its 
coding elements- has been shown to affect or interact with the learning process (Pederson 1987). As Clark 
(1983) sums it up, "We have moved from asking which medium was a better teacher to a concern with wliich 
* attributes* of media might combine with learner traits under different task conditions and perfonnance demands 
to produce different kinds of learning.** 

Tht CyberBuch program was pilot-tested by 44 second-year Gennan students at the Universiiy of 
California, Santa Barbara in November 1993. We tested tliree separate but related as|)ects of the program and 
its effectiveness: usability, learning behavior, effects on language learning, 

L Usability: We first assessed the usability of tlie program, with particular regard lo the multimedia 
components of the program based on questionnaires students filled out al'ter they Iiad worked with tiie prognun 
for two 50-minute class periods. There were a total of 42 questions, ranging from ease of use of llie program, to 
the clarity and effectiveness of the multimedia links, artd the users* perceptions of their own learning siyles. 
The most relevant questions for this paper involve student opinions of the various links and perceptions aboui 
how tliey leani best. 

Figure 1 shows that an overwhelming percentage (95% and 92%, respectively) agreed or strongly agreed 
that 'The still images [picture links] clearly showed what the word means** and that *The movies clearly 
showed what tlie word means.** The reactions to the statement 'The Gennan definitions [text links] for the 
word were clear and understandable" were mixed, with roughly a third (30%) agreeing or strongly agreeing, a 
tliird (32%) uncertain and a liiird (35%) disagreeing or strongly disagreeing. However, according to siudenis' 
verbal and written comments beyond the questionnaire, tliis is due not to the type of link, i.e. textual, hut ratlier 
to the difficulty of understanding definitions in German. 
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Fig. I. Clarity of links 



Fig. 2. Helpfulness of links for leaniin^ 



Similarly, Figure 2 shows tliat an ovenvhelming percentage (95%) agreed or strongly agreed that "Seeing a 
photograph or movie [visuid link] hel|)ed to reinforce Ie;iniing.** However, only about half (51%) agreed or 
strongly agreed that "Hearing tlie words spoken [audio Hnk] hel{>ed lo reinforce learning/* 

Students were also asked lo rank the different media links in order'of which hclj>ed tliem lo Iciuii reinemhcr 
individual vcKabulary words best, with 1 = helps most, 4 = helps leasl. Tlie stnking results aje iliat nciuly half 
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(43%) of llic students ranked pictures as helping die most aiid llie other half (46%) ranked movies as helping the 
most, while 51% ranked textual definitions as ilie third most helpful link ajid 68% ranked hearing the word 
spoken as ihc fourth most useful link (see Figures 3 ajid 4). 
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Fig. 3. Ranking of most useful links 



Fig. 4. Ranking of least useful links 



2. Learner Behavior: Secondly, we tracked learner beha\'ior while they used tlie program. h>pothesizing tliat 
the picture and movie links would appeal to students and they would choose such links to a greater degrce^'than 
"typical" textual or definitional links. The logs recorded provide data about which hnks were chosen and wliich 
ones were not used at all for both days tliat the program was used. 

After learners had used the program for two class periods, a '^surprise" vocabulary test was given, 
consisting of 17 words. The words were selected to represent the different multimedia links: 6 of them had 
only text links (definitions in German), 5 had only picture + text links, 5 had only movie + text links, and I had 
all three links (picture + movie + text). It should be noted that the audio component was included by default 
with all of the links. In tliis paper, we are restricting ourselves to the log data from these 17 words (ajid 
excluding the other 59 words in the program with links). 

The data show that when text links were available (17 words x 44 students = 748 available links), 38% of 
these links were chosen. Wliile iliere were fewer available picture and movie links (6 x 44 = 264 for each t)'|>e), 
they were chosen to a much greater degree, 58% and 74%, respectively. 

Arrow shows number of available links for each media type . 
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Fig. 5. Distribution of text links chosen by individual students 
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Fig. 6. Distribution of picture links chosen by individual students 
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Figures 5-7 show the distribution of all the media links chosen by individual students^ including multiple 
selection of a link. The arrows show the niunber (percentage) of available links for each type. 
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Fig. 7. Distribution of movie links chosen by individual students 

In addition to tracking the chosen links for these 17 words, we asked learners on their vocabulary quizzes 
to specify which link they thought helped them learn the word. We then compared tliese **self-reported" data to 
the logs which recorded actual behavior and found a significant dependence between the type of link chosen and 
whether learners reported that that particular type of link was useful. For the text links, there was a negative 
relationship (Pearson (1) = 51.67, p < 0.05), i.e.. when text links were chosen, studeuis named a nonAcxi link 
as being most helpful in 53% of the cases. In less than half of tlie cases (47%) a text link was found to be 
useful. On the other hand, for the picture and movie links, there was a positive relationship for both types of 
links between the actual choice of a link and reporting that it was useful. For picture links, Pearson (1) = 
20.26, p < 0.05, i.e., when picture links were chosen, students reported diat the picture link was helpful in 57% 

of tlie cases; for movie links, Pearson (1) = 26.85. p < 0.05, i.e., when move links were chosen, students 
stated that the movie link was helpful in 74% of the cases. 

3* Effects on Language Learning: Thirdly, we assessed the effectiveness of the program for two levels of 
language learning: short-term recall of vocabulary items and overall reading comprehension. We first wanted 
to determine whether there are any effects of the multimedia links on the learning of individual vocabulary 
words. A vocabulary test of 17 words, as described in Section 2. above, was administered to the students alter 
they had used the program for two 50-iiiinute class periods. Students were not forewarned that they would be 
tested, so they were not consciously trying to learn new words. The same test was administered after two 
weeks, again without telling the students that they would be re-tested. Results of both tests, showing the 
number of correct answers, are presented in Figure 8. Test words are furtlicr categorized as to the link wliich 
students reported as having helped them learn the word. It should be noted that of the possible number of 
answers to the vocabulciry test (17 words x 44 students = 74«), students made only 331 and 293 entries, 
resi)ectively» on the two tests. 
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Fig. 8. Correct answers out of total number of entries on vocabulary tests 



138 



Statistical tests to determine the correlation between the multimedia links chosen for each word and 
whether tlie word was learned correctly did not prove significant (see Table 1). However, a positive 
relationship was found for all types of links which could be chosen. When text links were chosen for words 
which had such links available and an answer was given on the first vocabulary test, 77% of the words were 
learned correctly and only 23% of the answers were incorrect. When picture links were chosen for words wliich 
had picture links, 75% of the answers were correct, and similarly, for the words with movie links, 79% were 
correct. 



Table 1. Correlation between media link chosen and vocabulary test results 



Vocabulary 


Text 


Picture 


Movie 


test result 


chosen 


not chosen 


chosen 


not chosen 


chosen 


not chosen 


total wrong 


47 


35 


13 


18 


5 


16 


total right 


132 


117 


60 


54 


15 


60 


pet. wrong 


14.2% 


.10.6% 


9.0% 


12.4% 


5.2% 


16.7% 


pel. ri^tht 


39.9% 


35.4% 


41.4% 


37.2% 


15.6% 


62.5% 




N=179 


N=152 


N=73 


N=72 


N=20 


N=76 




Pearson (0 


= 0.460, 


Pearson (1) 


= 1.115, 


Pearson y}- (1) 


= 0.144, 




p =0.497 




p =0.291 




p =0.704 





Striking is the fact that in the second "surprise'* vocabulary test two weeks later, almost tlie same total 
number of correct answers were given: 249 immediately after using tlie program for 2 days, and 235 two weeks 
laler. Broken down by type of link, there is in fact a slight improvement in tlie percentages: of the words wiili 
text links, 83% of tlie answers given were correct; of the words with picture links, 90% were correct; of the 
words witli movie links, 82% were correct. 

A significant dependence was found between the self-reported data from the users when tlie text link was 
reported as most helpful in learning a word and whether tlie word was actually correct on the vocabulary icsl. 
The Pearson x^ (1) = 3.89, p < 0.05, i.e., when students reported tliat a text link had been most useful for 
remembering a word, their answers on the vocabulary test for that word were correct in 68% of the cases. 
While no significant dependence was found for the cases where picture or movie links were thought to be most 
helpful, there was a positive relationship in both cases, and similar percentages of correct answers were given: 
75% and 83%, respectively. 

In order to test overall reading comprehension, students were asked to write a "recall protocol" of the story. 
This technique of summarizing a text in one's native language is widely used in assessing reading 
comprehension in both first and second language acquisition. The hypotliesis was that the QuickTime movie 
preview would provide students witli a global meaning of the text and that tlie oilier features, such as the movie- 
text link, which allowed them to hear the entire text read by native speakers, would also contribute to 
understanding. 

However, ilie pilot test results of the recall protocols showed ver>' poor comprehension. These results can 
be attributed to two main factors: first, the actual content of the preview, while aesthetically appealing, was 
much too subtle, and second, tliere were some teclmicaJ limitations. These limitations revolved around the fact 
that the movie (2:45 niin.) was too large to load on each student's hard drive and therefore had to be shown to 
each class as a group at the beginning of the hour. Students were tiot able to watch the movie ag*iin, 
individually, whenever tliey chose to. In addition, no headphones were available to students, so whenever a 
student chose a link or played the entire narration of the text, everyone else in the room would hear it, Tliis may 
have deterred students from using the text-movie link feature, and in fact, only a few ventured to use tliis feature 
at all. 

The preview is currently being redone and subsequent testing of the assumption that only knowing the 
meaning of individual words does not imply or insure understanding of a text will be conducted. Willi a more 
explicit preview, the effects of multimedia infoniialion on tlie higher-level process of comprehension of tlie text 
as a whole can then be assessed. 

Summary and Conclusions 

In summary, in assessing the effectiveness of our multimedia software in improving readiu" 
comprehension, we first investigated the usability of the program in temis of clarity of the different links (lexr. 
picture, movie). Users found the text links (definitions in German) ilie least helpful, probably due to the 
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difficulties in understanding die Gennan rather than to the type of link. On the oUier hand, picture and movie 
links were ranked as the most helpful types of links. (In the next testing phase, we will both improve the 
definitions in Gennan as well as add a second "layer'' of help, consisting of English descnpUons or iranslatious 

of the words or phrases.) ^ . , i 

We then examined the data from the user logs to determine which links were chosen and wheUier Uiere was 
a correlation between which links were actually chosen and which were reported to be most helpful. A 
statistically significant dependence was found for all three types of links. In the case of text links, there was a 
negative relationship: when a text link was chosen, users did not find Uiis link helpful in more than 50% of the 
cases. However, in the case of picture and movie links, when either of these links was chosen, users did find tlic 
link helpful in 57% and 74% of the cases, respectively. 

The data from the two vocabulary tests, administered immediately after the program had been used for two 
class periods and then again two weeks later, show, first of all, very litUe attrition of correct answers after Uie 
two week interval, and secondly, a positive, although not significant, correlation between the type of link chosen 
and whether tlie word was learned correctly. When text, picture or movie links were chosen for words wluch 
had such links available, high percentages of correct answers were given (77%, 75%, and 79%, respectively). 

In conclusion, students showed a definite preference for the picture and movie links (over the textual and 
audio links), both in their responses on the questionnaires about which links were most helpful, as well as in 
terms of which links they actually chose when using the program. While the movies were found to be most 
useful, the QT movie was, compared to the physical clarity of the textual and picture Imks, ol much lower 
quality. This suggests that although the quality of the links was uneven across types of link§, the more 
important factor appears to be type of link. r , • 

In addition this is supported by 1) the statistically significant /?(7j/7ive dependence of chosen picture and 
movie links with students reporting that these two types of links were helpful, and 2) the stauslically sigmlic;mt 
negative dependence when text links were chosen and students reported them as useful. 

In terms of the effectiveness of the different types of links for learning vocabulary words, there was a 
positive though not significant, relationship between type of link chosen and whether the word was learned 
correctly. A possible contributing factor which was not included in Uiis analysis is the fact that the program 
allows multiple links to be chosen in any combination, and Uierefore attributing learning to a single type of hnk 
for each word may not be warraiUed. Future testing will address this issue. 

While this pilot test only suggests that visual types of links or infonnation may be preferable to texiiuil 
links future tests will examine the hypothesis that students with different learning styles, e.g. visual, auditory, 
or textual, prefer different media types of information. If this hypothesis is then connmied, we could consider 
this to be strong support for the notion that multimedia systems bring a new dimension to educational soitwjire 
because tliey can be adapted to individual learning types. 
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